# Dagster Instance

<div className="border px-4 py-4 space-y-8">
  <div className="border px-4 py-4 space-y-4">
    <div>Storages</div>
    <div className="grid grid-cols-3 gap-8 justify-between">
      <InstanceDiagramBox href="#run-storage">Run Storage</InstanceDiagramBox>
      <InstanceDiagramBox href="#event-log-storage">Event Storage</InstanceDiagramBox>
      <InstanceDiagramBox href="#schedule-storage">Schedule Storage</InstanceDiagramBox>
    </div>
  </div>
  <div className="grid grid-cols-3 gap-8 justify-between">
    <div className="col-span-3 border px-4 py-4 space-y-4">
      <div>Daemons</div>
      <div className="grid grid-cols-2 gap-8 justify-between">
        <InstanceDiagramBox href="#scheduler">Scheduler</InstanceDiagramBox>
        <InstanceDiagramBox href="#run-coordinator">Run Coordinator</InstanceDiagramBox>
      </div>
    </div>
    <InstanceDiagramBox href="#run-launcher">Run Launcher</InstanceDiagramBox>
    <InstanceDiagramBox href="#compute-log-storage">Compute Log Storage</InstanceDiagramBox>
    <InstanceDiagramBox href="#local-artifact-storage">Local Artifact Storage</InstanceDiagramBox>
  </div>
</div>

## Overview

The <PyObject module="dagster" object="DagsterInstance" displayText="DagsterInstance" />
defines all of the configuration that Dagster needs for a single deployment - for example,
where to store the history of past runs and their associated logs, where to stream the raw logs
from solid compute functions, how to store local artifacts on disk, and how to launch new runs.

All of the processes and services that make up your Dagster deployment should share a single instance
config file so that they can effectively share information.

<Warning>
  Some important configuration, like execution parallelism, is also set on a
  per-pipeline-run basis rather than on the instance. See{" "}
  <Link href="/TODO">Pipeline Runs</Link> for pipeline configuration.
</Warning>

## Default local behavior

When you launch a Dagster process, like Dagit or the Dagster CLI commands, Dagster attempts to load
your instance. If the environment variable `DAGSTER_HOME` is set, Dagster will look for an
instance config file at `$DAGSTER_HOME/dagster.yaml`. This file contains each of the configuration
settings that make up the instance.

By default (if `dagster.yaml` is not present or nothing is specified in that file), Dagster will
store this information on the local filesystem, laid out like this:

    $DAGSTER_HOME
    ├── dagster.yaml
    ├── history
    │   ├── runs
    │   │   ├── 00636713-98a9-461c-a9ac-d049407059cd.db
    │   │   └── ...
    │   └── runs.db
    └── storage
        ├── 00636713-98a9-461c-a9ac-d049407059cd
        │   └── compute_logs
        │       ├── my_solid.compute.complete
        │       ├── my_solid.compute.err
        │       ├── my_solid.compute.out
        │       └── ...
        └── ...

The `runs.db` and `{run_id}.db` files are SQLite database files recording information about pipeline
runs and per-run event logs respectively. The `compute_logs` directories (one per
pipeline run) contain the stdout and stderr logs from the execution of the compute functions of each
solid in a pipeline.

If `DAGSTER_HOME` is not set, the Dagster tools will use an ephemeral instance for execution. In
this case, the run and event log storages will be in-memory rather than persisted to disk, and
filesystem storage will use a temporary directory that is cleaned up when the process exits. This is
useful for tests and is the default when invoking Python APIs such as <PyObject module="dagster"
object="execute_pipeline" displayText="execute_pipeline" /> directly.

## Instance Configuration YAML

In persistent Dagster deployments, you will typically want to configure many of the components on
the instance. For example, you may want to use a Postgres instance to store runs and the corresponding
event logs, and to stream compute logs to an S3 bucket.

To do this, provide a `$DAGSTER_HOME/dagster.yaml` file. Dagit and all Dagster tools will look for
this file on startup. In the `dagster.yaml` file, you can configure many different aspects of
your Dagster Instance, all of which are detailed below.

Note that Dagster supports retrieving instance YAML values from environment
variables, using an `env:` key instead of a string literal value. Examples of using `env:` are
included in the sample configurations below:

### Run Storage

This configures how the history of runs is persisted - this includes metadata on a given run's status,
relevant timestamps, and other useful data to inspect a run's history.

To configure Run Storage, you should set the `run_storage` attribute in your `dagster.yaml`. There
are _three_ available options:

#### SqliteRunStorage (Default)

<PyObject module="dagster.core.storage.runs" object="SqliteRunStorage" /> uses a Sqlite DB as the backing run storage.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_run_storage_sqlite endbefore=end_marker_run_storage_sqlite
# there are two ways to set run_storage to SqliteRunStorage

# this config manually sets the directory (`base_dir`) for Sqlite to store run information in:
run_storage:
  module: dagster.core.storage.runs
  class: SqliteRunStorage
  config:
    base_dir: /path/to/dir

# and this config grabs the directory from an environment variable
run_storage:
  module: dagster.core.storage.runs
  class: SqliteRunStorage
  config:
    base_dir:
      env: SQLITE_RUN_STORAGE_BASE_DIR
```

#### PostgresRunStorage

<PyObject module="dagster_postgres" object="PostgresRunStorage"/> uses a Postgres DB as the backing run storage solution.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_run_storage_postgres endbefore=end_marker_run_storage_postgres
# There are three ways to set run_storage to PostgresRunStorage

# this config manually sets the Postgres credentials
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username: { DAGSTER_PG_USERNAME }
      password: { DAGSTER_PG_PASSWORD }
      hostname: { DAGSTER_PG_HOSTNAME }
      db_name: { DAGSTER_PG_DB }
      port: 5432

# and this config grabs the database credentials from environment variables
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_PG_USERNAME
      password:
        env: DAGSTER_PG_PASSWORD
      hostname:
        env: DAGSTER_PG_HOST
      db_name:
        env: DAGSTER_PG_DB
      port: 5432

# and this config sets the credentials via DB connection string / url:
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_url: { PG_DB_CONN_STRING }

# This config gets the DB connection string / url via environment variables:
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_url:
      env: PG_DB_CONN_STRING
```

#### MySQLRunStorage

<PyObject module="dagster_mysql" object="MySQLRunStorage"/> uses a MySQL DB as the backing run storage solution.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_run_storage_mysql endbefore=end_marker_run_storage_mysql
# There are three ways to set run_storage to MySQLRunStorage

# this config manually sets the MySQL credentials
run_storage:
  module: dagster_mysql.run_storage
  class: MySQLRunStorage
  config:
  mysql_db:
    username: { DAGSTER_MYSQL_USERNAME }
    password: { DAGSTER_MYSQL_PASSWORD }
    hostname: { DAGSTER_MYSQL_HOSTNAME }
    db_name: { DAGSTER_MYSQL_DB }
    port: 3306


# and this config grabs the database credentials from environment variables
run_storage:
  module: dagster_mysql.run_storage
  class: MySQLRunStorage
  config:
    mysql_db:
      username:
        env: DAGSTER_MYSQL_USERNAME
      password:
        env: DAGSTER_MYSQL_PASSWORD
      hostname:
        env: DAGSTER_MYSQL_HOSTNAME
      db_name:
        env: DAGSTER_MYSQL_DB
      port: 3306

# and this config sets the credentials via DB connection string / url:
run_storage:
  module: dagster_mysql.run_storage
  class: MySQLRunStorage
  config:
    mysql_url: { MYSQL_DB_CONN_STRING }

# this config grabs the MySQL connection string from environment variables
run_storage:
  module: dagster_mysql.run_storage
  class: MySQLRunStorage
  config:
    mysql_url:
      env: MYSQL_DB_CONN_STRING
```

### Event Log Storage

This controls how the structured event logs produced by each run are persisted - this includes events
related to solids starting/running/completing, pipeline events, and useful event-related data.

To configure Event Log Storage, you should set the `event_log_storage` attribute in your `dagster.yaml`.
There are _four_ available options:

#### SqliteEventLogStorage (Default)

<PyObject module="dagster.core.storage.event_log" object="SqliteEventLogStorage" /> uses a Sqlite database as the backing storage for event logs.
However, it utilizes a seperate Sqlite database (i.e. directory) for each run.
In other words, the storage is _run-sharded_. This means that certain useful features, like the Assets page,
are not usable since Dagster can't perform cross-run queries.

Note that by leaving your `event_log_storage` attribute blank, Run Storage will be set to Sqlite by default.
If you would like to manually specify `event_log_storage`, add the following snippet to your `dagster.yaml`:

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_event_log_storage_sqlite endbefore=end_marker_event_log_storage_sqlite
# there are two ways to set `event_log_storage` to SqliteEventLogStorage

# the first manually sets the directory (`base_dir`) to write event log data to:
event_log_storage:
  module: dagster.core.storage.event_log
  class: SqliteEventLogStorage
  config:
    base_dir: /path/to/dir

# and the second grabs the directory from an environment variable
event_log_storage:
  module: dagster.core.storage.event_log
  class: SqliteEventLogStorage
  config:
    base_dir:
      env: SQLITE_EVENT_LOG_STORAGE_BASE_DIR
```

#### ConsolidatedSqliteEventLogStorage <Experimental/>

<PyObject module="dagster.core.storage.event_log" object="ConsolidatedSqliteEventLogStorage" /> is meant to imitate the behavior of Postgres or MySQL-based event log storages,
as it is not _run sharded_. That is, all runs are persisted to the same Sqlite DB, which allows for cross-run
queries (& thus use of certain Dagster features, like the Assets page).

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_event_log_storage_consolidated_sqlite endbefore=end_marker_event_log_storage_consolidated_sqlite
# there are two ways to set `event_log_storage` to ConsolidatedSqliteEventLogStorage

# the first manually sets the directory (`base_dir`) to write event log data to:
event_log_storage:
  module: dagster.core.storage.event_log
  class: ConsolidatedSqliteEventLogStorage
  config:
    base_dir: /path/to/dir

# and the second grabs the directory from an environment variable
event_log_storage:
  module: dagster.core.storage.event_log
  class: ConsolidatedSqliteEventLogStorage
  config:
    base_dir:
      env: CONSOLIDATED_SQLITE_EVENT_LOG_STORAGE_BASE_DIR
```

#### PostgresEventLogStorage

<PyObject module="dagster_postgres" object="PostgresEventLogStorage" /> utilizes a Postgres Database as the backing Event Log Storage solution.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_event_log_storage_postgres endbefore=end_marker_event_log_storage_postgres
# There are four ways to set event_log_storage to PostgresEventLogStorage

# this config manually sets the Postgres credentials
event_log_storage:
  module: dagster_postgres.event_log_storage
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username: { DAGSTER_PG_USERNAME }
      password: { DAGSTER_PG_PASSWORD }
      hostname: { DAGSTER_PG_HOSTNAME }
      db_name: { DAGSTER_PG_DB }
      port: 5432

# and this config grabs the database credentials from environment variables
event_log_storage:
  module: dagster_postgres.event_log_storage
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_PG_USERNAME
      password:
        env: DAGSTER_PG_PASSWORD
      hostname:
        env: DAGSTER_PG_HOST
      db_name:
        env: DAGSTER_PG_DB
      port: 5432

# and this config sets the credentials via DB connection string / url:
event_log_storage:
  module: dagster_postgres.event_log_storage
  class: PostgresEventLogStorage
  config:
    postgres_url: { PG_DB_CONN_STRING }

# This config gets the DB connection string / url via environment variables:
event_log_storage:
  module: dagster_postgres.event_log_storage
  class: PostgresEventLogStorage
  config:
    postgres_url:
      env: PG_DB_CONN_STRING
```

#### MySQLEventLogStorage

<PyObject module="dagster_mysql" object="MySQLEventLogStorage" /> uses a MySQL DB as the backing event log storage solution.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_event_log_storage_mysql endbefore=end_marker_event_log_storage_mysql
# There are four ways to set event_log_storage to MySQLRunStorage

# this config manually sets the MySQL credentials
event_log_storage:
  module: dagster_mysql.event_log_storage
  class: MySQLEventLogStorage
  config:
  mysql_db:
    username: { DAGSTER_MYSQL_USERNAME }
    password: { DAGSTER_MYSQL_PASSWORD }
    hostname: { DAGSTER_MYSQL_HOSTNAME }
    db_name: { DAGSTER_MYSQL_DB }
    port: 3306


# and this config grabs the database credentials from environment variables
event_log_storage:
  module: dagster_mysql.event_log_storage
  class: MySQLEventLogStorage
  config:
    mysql_db:
      username:
        env: DAGSTER_MYSQL_USERNAME
      password:
        env: DAGSTER_MYSQL_PASSWORD
      hostname:
        env: DAGSTER_MYSQL_HOSTNAME
      db_name:
        env: DAGSTER_MYSQL_DB
      port: 3306

# and this config sets the credentials via DB connection string / url:
event_log_storage:
  module: dagster_mysql.event_log_storage
  class: MySQLEventLogStorage
  config:
    mysql_url: { MYSQL_DB_CONN_STRING }

# this config grabs the MySQL connection string from environment variables
event_log_storage:
  module: dagster_mysql.event_log_storage
  class: MySQLEventLogStorage
  config:
    mysql_url:
      env: MYSQL_DB_CONN_STRING
```

### Scheduler

This provides an optional scheduler which controls execution of pipeline runs at regular intervals.

To configure the scheduler, set `scheduler` in your `dagster.yaml`. If you don't set `scheduler`,
no Scheduler is configured by default.

There are _three_ options for the Scheduler: <PyObject module="dagster.core.scheduler" object="DagsterDaemonScheduler" />, <PyObject module="dagster_cron.cron_scheduler" object="SystemCronScheduler" />, and <PyObject module="dagster_k8s" object="K8sScheduler" />.
**We recommend using `DagsterDaemonScheduler`**.

<Warning>
  <PyObject module="dagster_cron.cron_scheduler" object="SystemCronScheduler" /> and <PyObject module="dagster_k8s" object="K8sScheduler" />
  are deprecated and will be removed in the 0.12.0 Dagster release.
</Warning>

#### DagsterDaemonScheduler (Default)

<PyObject module="dagster.core.scheduler" object="DagsterDaemonScheduler" /> submits runs from the dagster-daemon (long-lived) process.
It periodically checks each running schedule for execution times that don’t have runs yet and launches them.

Please also see the <Link href="/deployment/dagster-daemon">Daemon Deployment Docs</Link> for more context on deployment.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_dagster_daemon endbefore=end_marker_dagster_daemon
scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler
```

### Schedule Storage

This option controls the backing storage used by the scheduler to manage the state of schedules (for example,
tracking `Queued` runs) and persist records of attempts-related data.

To configure Schedule Storage, you should set the `schedule_storage` key in your `dagster.yaml`. There
are _three_ available options:

#### SqliteScheduleStorage (Default)

<PyObject module="dagster.core.storage.schedules" object="SqliteScheduleStorage" /> uses a Sqlite DB as the backing schedule storage solution.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_schedule_storage_sqlite endbefore=end_marker_schedule_storage_sqlite
# there are two ways to set `schedule_storage` to SqliteScheduleStorage

# the first manually sets the directory (`base_dir`) to write schedule-related data to:
schedule_storage:
  module: dagster.core.storage.schedules
  class: SqliteScheduleStorage
  config:
    base_dir: /path/to/dir

# the second grabs the directory from an environment variable
schedule_storage:
  module: dagster.core.storage.schedules
  class: SqliteScheduleStorage
  config:
    base_dir:
      env: SQLITE_SCHEDULE_STORAGE_DIRECTORY
```

#### PostgreScheduleStorage

<PyObject module="dagster_postgres" object="PostgresScheduleStorage" /> uses a Postgres DB as the backing schedule storage solution.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_schedule_storage_postgres endbefore=end_marker_schedule_storage_postgres
# There are three ways to set schedule_storage to PostgresScheduleStorage

# this config manually sets the Postgres credentials
schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username: { DAGSTER_PG_USERNAME }
      password: { DAGSTER_PG_PASSWORD }
      hostname: { DAGSTER_PG_HOSTNAME }
      db_name: { DAGSTER_PG_DB }
      port: 5432

# and this config grabs the database credentials from environment variables
schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_PG_USERNAME
      password:
        env: DAGSTER_PG_PASSWORD
      hostname:
        env: DAGSTER_PG_HOST
      db_name:
        env: DAGSTER_PG_DB
      port: 5432

# and this config sets the credentials via DB connection string / url:
schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_url: { PG_DB_CONN_STRING }

# This config gets the DB connection string / url via environment variables:
schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_url:
      env: PG_DB_CONN_STRING
```

#### MySQLScheduleStorage

<PyObject module="dagster_mysql" object="MySQLScheduleStorage" /> uses a MySQL DB as the backing schedule storage solution.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_schedule_storage_mysql endbefore=end_marker_schedule_storage_mysql
# There are three ways to set schedule_storage to MySQLScheduleStorage

# this config manually sets the MySQL credentials
schedule_storage:
  module: dagster_mysql.schedule_storage
  class: MySQLScheduleStorage
  config:
  mysql_db:
    username: { DAGSTER_MYSQL_USERNAME }
    password: { DAGSTER_MYSQL_PASSWORD }
    hostname: { DAGSTER_MYSQL_HOSTNAME }
    db_name: { DAGSTER_MYSQL_DB }
    port: 3306


# and this config grabs the database credentials from environment variables
schedule_storage:
  module: dagster_mysql.schedule_storage
  class: MySQLScheduleStorage
  config:
    mysql_db:
      username:
        env: DAGSTER_MYSQL_USERNAME
      password:
        env: DAGSTER_MYSQL_PASSWORD
      hostname:
        env: DAGSTER_MYSQL_HOSTNAME
      db_name:
        env: DAGSTER_MYSQL_DB
      port: 3306

# and this config sets the credentials via DB connection string / url:
schedule_storage:
  module: dagster_mysql.schedule_storage
  class: MySQLScheduleStorage
  config:
    mysql_url: { MYSQL_DB_CONN_STRING }

# this config grabs the MySQL connection string from environment variables
schedule_storage:
  module: dagster_mysql.schedule_storage
  class: MySQLScheduleStorage
  config:
    mysql_url:
      env: MYSQL_DB_CONN_STRING
```

### Run Launcher

This determines where runs are executed.

There are _two_ Dagster-provided options for the Run Launcher; users also can write custom run launchers.
Check out the <Link href="/deployment/run-launcher">Run Launcher page</Link> for more information.

To configure the Run Launcher, set `run_launcher` in your `dagster.yaml` in one of the following ways:

#### DefaultRunLauncher (Default)

The <PyObject module="dagster.core.launcher" object="DefaultRunLauncher" /> spawns a new process in the same node as a pipeline's repository location.
Please see the <Link href="/deployment/run-launcher#default-run-launcher">Run Launcher Docs</Link> for deployment information.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_run_launcher_default endbefore=end_marker_run_launcher_default
run_launcher:
  module: dagster.core.launcher
  class: DefaultRunLauncher
```

#### K8sRunLauncher

The <PyObject module="dagster_k8s" object="K8sRunLauncher" /> allocates a Kubernetes Job per run.
Please see the <Link href="/deployment/run-launcher#other-run-launchers">Run Launcher Docs</Link> for deployment information.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_run_launcher_k8s endbefore=end_marker_run_launcher_k8s
# there are multiple ways to configure the K8sRunLauncher

# you can set the follow configuration values directly
run_launcher:
  module: dagster_k8s.launcher
  class: K8sRunLauncher
  config:
    service_account_name: pipeline_run_service_account
    job_image: my_project/dagster_image:latest
    instance_config_map: dagster-instance
    postgres_password_secret: dagster-postgresql-secret

# alternatively, you can grab any of these config values from environment variables:
run_launcher:
  module: dagster_k8s.launcher
  class: K8sRunLauncher
  config:
    service_account_name:
      env: PIPELINE_RUN_SERVICE_ACCOUNT
    job_image:
      env: DAGSTER_IMAGE_NAME
    instance_config_map:
      env: DAGSTER_INSTANCE_CONFIG_MAP
    postgres_password_secret:
      env: DAGSTER_POSTGRES_SECRET
```

### Run Coordinator

This determines the policy used to determine the prioritization rules nad concurrency limits for runs.
Please see the <Link href="/deployment/run-coordinator">Run Coordinator Docs</Link> for more information and for troubleshooting help.

To configure the Run Coordinator, set the `run_coodinator` key in your `dagster.yaml`. There are _two_ options:

#### DefaultRunCoordinator (Default)

The <PyObject module="dagster.core.run_coordinator" object="DefaultRunCoordinator" /> immediately sends runs to the run launcher (no notion of `Queued` runs).

See the <Link href="/deployment/run-coordinator#defaultruncoordinator">Run Coordinator Docs</Link> for more information.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_run_coordinator_default endbefore=end_marker_run_coordinator_default
# Since DefaultRunCoordinator is the default option, omitting the `run_coordinator` key will also suffice,
# but if you would like to set it explicitly:
run_coordinator:
  module: dagster.core.run_coordinator
  class: DefaultRunCoordinator
```

#### QueuedRunCoordinator

The <PyObject module="dagster.core.run_coordinator" object="QueuedRunCoordinator" /> allows you to set limits on the number of runs that can be executing at once.
This sends runs to a Dequeuer process (i.e. a Scheduler / Daemon) via run storage. Note that this requires
a daemon process to be active to actually launch the runs.

This option has several configuration possibilities which allow for both limit the overall number of concurrent
runs and more specific limits based on run tags - e.g. perhaps only 4 runs interacting with a
cloud service should run concurrently to avoid being throttled.

For more information, see the <Link href="/deployment/run-coordinator#queuedruncoordinator">Run Coordinator Docs</Link>.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_run_coordinator_queued endbefore=end_marker_run_coordinator_queued
# There are a few ways to configure the QueuedRunCoordinator:

# this first option has concurrency limits set to default values
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

# this second option manually specifies limits:
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 25
    tag_concurrency_limits:
      - key: "database"
        value: "redshift"
        limit: 4
      - key: "dagster/backfill"
        limit: 10

# as always, some or all of these values can be obtained from environment variables:
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs:
      env: DAGSTER_OVERALL_CONCURRENCY_LIMIT
    tag_concurrency_limits:
      - key: "database"
        value: "redshift"
        limit:
          env: DAGSTER_REDSHIFT_CONCURRENCY_LIMIT
      - key: "dagster/backfill"
        limit:
          env: DAGSTER_BACKFILL_CONCURRENCY_LIMIT
```

### Compute Log Storage

Compute log storage controls the capture and persistence of raw stdout & stderr text logs.

To configure Compute Log Storage, set the `compute_logs` key in your `dagster.yaml`.

#### LocalComputeLogManager (Default)

<PyObject module="dagster.core.storage.local_compute_log_manager" object="LocalComputeLogManager" /> writes stdout & stderr logs to disk.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_compute_log_storage_local endbefore=end_marker_compute_log_storage_local
# there are two ways to set the directory that the LocalComputeLogManager writes
# stdout & stderr logs to

# You could directly set the `base_dir` key
compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir: /path/to/directory

# Alternatively, you could set the `base_dir` key to an environment variable
compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir:
      env: LOCAL_COMPUTE_LOG_MANAGER_DIRECTORY
```

#### AzureBlobComputeLogManager

<PyObject module="dagster_azure.blob" object="AzureBlobComputeLogManager" /> writes stdout & stderr logs to Azure Blob Storage.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_compute_log_storage_blob endbefore=end_marker_compute_log_storage_blob
# there are multiple ways to configure the AzureBlobComputeLogManager

# you can set the necessary configuration values directly:
compute_logs:
  module: dagster_azure.blob.compute_log_manager
  class: AzureBlobComputeLogManager
  config:
    storage_account: mycorp-dagster
    container: compute-logs
    secret_key: foo
    local_dir: /tmp/bar
    prefix: dagster-test-

# alternatively, you can obtain any of these config values from environment variables
compute_logs:
  module: dagster_azure.blob.compute_log_manager
  class: AzureBlobComputeLogManager
  config:
    storage_account:
      env: MYCORP_DAGSTER_STORAGE_ACCOUNT_NAME
    container:
      env: CONTAINER_NAME
    secret_key:
      env: SECRET_KEY
    local_dir:
      env: LOCAL_DIR_PATH
    prefix:
      env: DAGSTER_COMPUTE_LOG_PREFIX
```

#### S3ComputeLogManager

<PyObject module="dagster_aws.s3" object="S3ComputeLogManager" /> writes stdout & stderr logs to AWS S3.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_compute_log_storage_s3 endbefore=end_marker_compute_log_storage_s3
# there are multiple ways to configure the S3ComputeLogManager

# you can set the config values directly:
compute_logs:
  module: dagster_aws.s3.compute_log_manager
  class: S3ComputeLogManager
  config:
    bucket: "mycorp-dagster-compute-logs"
    prefix: "dagster-test-"

# or grab some or all of them from environment variables
compute_logs:
  module: dagster_aws.s3.compute_log_manager
  class: S3ComputeLogManager
  config:
    bucket:
      env: MYCORP_DAGSTER_COMPUTE_LOGS_BUCKET
    prefix:
      env: DAGSTER_COMPUTE_LOG_PREFIX
```

### Local Artifact Storage

The local artifact storage is used to configure storage for any artifacts that require a local disk,
such as schedules, or when using the filesystem system storage to manage files and intermediates.

Note that pipeline intermediates storage is itself not configured this way, but instead is configured
on the pipeline level (i.e. via <Link href="/concepts/io-management/io-managers">IO Managers</Link>)

To configure Local Artifact Storage, set `local_artifact_storage` as follows in your `dagster.yaml`:

#### LocalArtifactStorage (Default)

<PyObject module="dagster.core.storage.root" object="LocalArtifactStorage" /> is currently the only option for Local Artifact Storage.
This configures the directory used by the default filesystem intermediates storage, as well as
any schedule-related artifacts that require a local disk.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_local_artifact_storage endbefore=end_marker_local_artifact_storage
# there are two possible ways to configure LocalArtifactStorage

# example local_artifact_storage setup pointing to /var/shared/dagster directory
local_artifact_storage:
  module: dagster.core.storage.root
  class: LocalArtifactStorage
  config:
    base_dir: "/path/to/dir"

# alternatively, `base_dir` can be set to an environment variable
local_artifact_storage:
  module: dagster.core.storage.root
  class: LocalArtifactStorage
  config:
    base_dir:
      env: DAGSTER_LOCAL_ARTIFACT_STORAGE_DIR
```

### Telemetry

This allows opting in/out (set to `true` by default) of Dagster collecting
anonymized usage statistics.

To configure Telemetry, set the `telemetry` key in your `dagster.yaml`.

For more information on how and why we use telemetry, please visit the <Link href="/getting-started/telemetry">Telemetry Docs</Link>.

```yaml file=/deploying/dagster_instance/dagster.yaml startafter=start_marker_telemetry endbefore=end_marker_telemetry
# Allows opting out of Dagster collecting usage statistics.
telemetry:
  enabled: false
```
